Automatically allocating clients for software program testing

ABSTRACT

Techniques are described herein that are capable of automatically allocating clients for testing a software program. For instance, a number of the clients that are to be allocated for the testing may be determined based on a workload that is to be imposed by the clients during execution of the testing. For example, the number of the clients may be a minimum number of the clients that is capable of accommodating the workload. In accordance with this example, the minimum number of the clients may be allocated in a targeted environment so that the test may be performed on those clients. Additional clients may be allocated along with the minimum number of the clients in the targeted environment to accommodate excess workload.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.13/227,939, entitled “Automatically Allocating Clients for SoftwareProgram Testing,” filed Sep. 8, 2011, which is incorporated herein byreference in its entirety.

BACKGROUND

Software programs traditionally are tested using a variety of testsbefore the software programs are sold to end users. The purpose of suchtests usually is to simulate certain conditions to determine whether thesoftware programs are capable of operating under those conditions. Eachtest typically is written as a stand-alone test having its own code,which is separate from the code of the other tests. In fact, differentframeworks may be used for the different tests. A framework allowssoftware having a generic functionality to be selectively changed byuser code to provide application specific software. Writing andmaintaining separate code and/or using different frameworks for thevarious tests may consume substantial time and/or resources.

Moreover, each test often specifies a predetermined fixed number ofclients to be allocated for participation in the test. Restricting thenumber of clients in this manner hinders the use of the test for testingscenarios for which the test was not initially configured.

SUMMARY

Various approaches are described herein for, among other things,automatically allocating clients for testing a software program. Forinstance, a number of the clients that are to be allocated for thetesting may be determined based on a workload that is to be imposed bythe clients during execution of the testing. For example, the number ofthe clients may be a minimum number of the clients that is capable ofaccommodating the workload. In accordance with this example, the minimumnumber of the clients may be allocated in a targeted environment so thatthe test may be performed on those clients. Additional clients may beallocated along with the minimum number of the clients in the targetedenvironment to accommodate excess workload.

Some of the approaches described herein perform multi-modal testing ofthe software program. In accordance with these approaches, a single testis defined by code that is configurable to accommodate multiple types oftest scenarios. Accordingly, the functionality of multiple types oftests may be performed using respective modes of the single test.Examples of modes include but are not limited to a functionality mode, aperformance mode, a stress mode, and a scalability mode. The test in thefunctional mode is configured to determine whether the software programgenerates an expected output. The test in the performance mode isconfigured to determine an amount of time that the software programtakes to perform an operation upon receipt of an instruction to performthe operation. The test in the stress mode is configured to determine acapability of the software program to remain operational in response toan increasing load over a period of time. The test in the scalabilitymode is configured to incrementally increase a workload that isassociated with the test and to determine, at each incremental increase,a number of clients that are to be allocated in a targeted environmentto perform operations that impose the workload against the targetedenvironment. Any one or more of the modes may have a predeterminedconfiguration. A configuration of any one or more of the modes may bedynamically determined (e.g., based on characteristic(s) of the testingenvironment).

A method is described in which a number of clients to be allocated in atargeted environment for participation in a test of a software programis automatically determined based on a specified cumulative workload.The number of clients in the targeted environment is automaticallyallocated to perform operations that impose the cumulative workloadagainst the targeted environment. Each client of the number of clientsimposes a respective proportion of the cumulative load against thetargeted environment. The test of the software program is executed withrespect to the number of clients such that each client of the number ofclients simulates use of the software program by at least one user.

A system is described that includes determination logic, allocationlogic, and execution logic. The determination logic is configured toautomatically determine a number of clients to be allocated in atargeted environment for participation in a test of a software programbased on a specified cumulative workload. The allocation logic isconfigured to automatically allocate the number of clients in thetargeted environment to perform operations that impose the cumulativeworkload against the targeted environment. Each client of the number ofclients imposes a respective proportion of the cumulative load againstthe targeted environment. The execution logic is configured to executethe test of the software program with respect to the number of clientssuch that each client of the number of clients simulates use of thesoftware program by at least one user.

A computer program product is described that includes acomputer-readable medium having computer program logic recorded thereonfor enabling a processor-based system to perform multi-modal testing ofa software program. The computer program product includes first, second,and third program logic modules. The first program logic module is forenabling the processor-based system to automatically determine a numberof clients to be allocated in a targeted environment for participationin a test of a software program for each of a plurality of modes of thetest based on a specified cumulative workload. The second program logicmodule is for enabling the processor-based system to automaticallyallocate the respective number of clients in the targeted environmentfor each mode to perform operations that impose the cumulative workloadagainst the targeted environment, each client of the number of clientsimposing a respective proportion of the cumulative load against thetargeted environment. The third program logic module is for enabling theprocessor-based system to execute the test of the software program withrespect to the number of clients such that executes the test in a firstmode of the plurality of modes with respect to a first number of theclients and that executes the test in a second mode of the plurality ofmodes with respect to a second number of the clients. The second numberis different from the first number. Each client of the first number ofclients and each client of the second number of clients simulates use ofthe software program by at least one user.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Moreover, itis noted that the invention is not limited to the specific embodimentsdescribed in the Detailed Description and/or other sections of thisdocument. Such embodiments are presented herein for illustrativepurposes only. Additional embodiments will be apparent to personsskilled in the relevant art(s) based on the teachings contained herein.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

The accompanying drawings, which are incorporated herein and form partof the specification, illustrate embodiments of the present inventionand, together with the description, further serve to explain theprinciples involved and to enable a person skilled in the relevantart(s) to make and use the disclosed technologies.

FIG. 1 is a block diagram of an example computer system in accordancewith an embodiment.

FIGS. 2-4 depict flowcharts of example methods for automaticallyallocating clients for software program testing in accordance withembodiments.

FIG. 5 is a block diagram of an example implementation of anauto-allocation tester shown in FIG. 1 in accordance with an embodiment.

FIG. 6 is a block diagram of an example schema that may be used todefine a structure of a database that is included in a store shown inFIG. 1 in accordance with an embodiment.

FIG. 7 illustrates an example client-side error log in accordance withan embodiment.

FIG. 8 illustrates an example server-side error log in accordance withan embodiment.

FIG. 9 depicts an example computer in which embodiments may beimplemented.

The features and advantages of the disclosed technologies will becomemore apparent from the detailed description set forth below when takenin conjunction with the drawings, in which like reference charactersidentify corresponding elements throughout. In the drawings, likereference numbers generally indicate identical, functionally similar,and/or structurally similar elements. The drawing in which an elementfirst appears is indicated by the leftmost digit(s) in the correspondingreference number.

DETAILED DESCRIPTION I. Introduction

The following detailed description refers to the accompanying drawingsthat illustrate exemplary embodiments of the present invention. However,the scope of the present invention is not limited to these embodiments,but is instead defined by the appended claims. Thus, embodiments beyondthose shown in the accompanying drawings, such as modified versions ofthe illustrated embodiments, may nevertheless be encompassed by thepresent invention.

References in the specification to “one embodiment,” “an embodiment,”“an example embodiment,” or the like, indicate that the embodimentdescribed may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Moreover, such phrasesare not necessarily referring to the same embodiment. Furthermore, whena particular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the relevant art(s) to implement suchfeature, structure, or characteristic in connection with otherembodiments whether or not explicitly described.

II. Example Embodiments

Example embodiments described herein are capable of automaticallyallocating clients for testing a software program. For instance, anumber of the clients that are to be allocated for the testing may bedetermined based on a workload that is to be imposed by the clientsduring execution of the testing. For example, the number of the clientsmay be a minimum number of the clients that is capable of accommodatingthe workload. In accordance with this example, the minimum number of theclients may be allocated in a targeted environment so that the test maybe performed on those clients. It will be recognized that additionalclients may be allocated along with the minimum number of the clients inthe targeted environment to accommodate excess workload.

Some example embodiments are capable of performing multi-modal testingof the software program. In accordance with these embodiments, a singletest is defined by code that is configurable to accommodate multipletypes of test scenarios. Accordingly, such embodiments are capable ofperforming the functionality of multiple types of tests using respectivemodes of the single test. Examples of modes include but are not limitedto a functionality mode, a performance mode, a stress mode, and ascalability mode. These example modes are described in further detailbelow with reference to FIG. 2.

The testing of the software program may simulate usage of the softwareprogram by users. Accordingly, the various modes of the test may beconfigured to simulate respective conditions that may be imposed by suchusers. Configuring the code that defines the test to accommodate a modemay involve changing parameters that are associated with the test. Forinstance, central processing unit (CPU) utilization, memory utilization,and/or a number of network connections that are utilized by the test maybe varied to accommodate the mode.

Example techniques described herein have a variety of benefits ascompared to conventional techniques for testing a software program. Forinstance, the example techniques may provide a software program-agnosticscalable framework which is capable of scaling up and down dynamicallybased on a workload that is imposed against a target environment. Forinstance, scaling the framework may involve changing a number of clientsthat are to participate in the test, changing the workload, changinginformation that is reported and/or a format thereof, etc. The exampletechniques may be capable of accommodating arbitrary (e.g., previouslyunknown) test scenarios. For instance, the techniques may be capable ofgenerating a targeted load for each of the test scenarios, measuringlatency and/or throughput of the software program, automaticallycollecting information regarding performance, scalability, availability,etc. of the software program, performing triage, and/or generatingreports that include the aforementioned information.

The example techniques may accelerate a delivery of a test to obtain arelatively greater test coverage with a relatively low cost, as comparedto conventional software program tests. Performance, race condition,and/or scalability problems regarding a software program may bediscovered earlier in the development cycle of the software program byusing the example techniques. Any suitable software program may betested using the example techniques. For instance, the software programmay be a cloud service. The example techniques may be capable of testingany type of cloud service and/or any size of cloud service.

The example techniques may consume less time and/or fewer resources thanthe conventional techniques. For instance, the example techniques maycollect test information only if the test information indicates afailure with respect to the software program. The example techniques maybe implemented on top of standard off-the-shelf cloud platformtechnology and/or cloud-based services.

FIG. 1 is a block diagram of an example computer system 100 inaccordance with an embodiment. Generally speaking, computer system 100operates to test software programs. As shown in FIG. 1, computer system100 includes a plurality of clients 102A-102M, a network 104, anauto-allocation program tester 106, a plurality of servers 108A-108N,and a store 110. Communication among clients 102A-102M, auto-allocationprogram tester 106, servers 108A-108N, and store 110 is carried out overnetwork 104 using well-known network communication protocols. Network104 may be a wide-area network (e.g., the Internet), a local areanetwork (LAN), another type of network, or a combination thereof.

Clients 102A-102M are processing systems that are capable ofcommunicating with servers 108A-108N. An example of a processing systemis a system that includes at least one processor that is capable ofmanipulating data in accordance with a set of instructions. Forinstance, a processing system may be a computer (e.g., a desktopcomputer, a laptop computer, a tablet computer, etc.), a personaldigital assistant, a cellular telephone, etc. Although clients 102A-102Mare described herein as being processing systems, it will be recognizedthat any one or more of clients 102A-102M may be implemented as avirtual machine.

Clients 102A-102M may be configured to perform client-side aspects ofsoftware programs (e.g., cloud services) that are to be tested inaccordance with the example embodiments described herein. Clients102A-102M are configured to provide requests 118A-118N (e.g., HTTPrequests) to servers 108A-108N for the purpose of accessing server-sideaspects of the software programs that are executed thereon. Examples ofa cloud service (a.k.a. a web application or a web service) include butare not limited to webmail (e.g., Gmail®, Yahoo!® mail, and MSN® mail),an online retail application, an online auction, a wiki, an onlinesocial networking service (e.g., Facebook®, LinkedIn®, MySpace®, orTwitter®), an online photo hosting service (e.g., Flickr® or Picasa®),an online office suite service (e.g., Google® Docs), an online datastorage service (e.g., Egnyte®, Amazon® S3, or Windows Azure®), etc. Forinstance, a user may initiate a request using a client (e.g., a Webbrowser, Web crawler, non-Web-enabled client, etc.) deployed on a client102 that is owned by or otherwise accessible to the user for the purposeof accessing server-side aspects of a software program via network 104.

Clients 102A-102M receive access to the server-side aspects of thesoftware programs that are identified in the respective requests112A-112M via respective connections 120A-120M and one or more ofconnections 122A-122N. For instance, client 102A receives access toserver-side aspects of software program(s) that are identified inrequest(s) 118A via connection 120A and one or more of connections122A-122N, depending on which of the servers 108A-108N execute theserver-side aspects of the software program(s); client 102B receivesaccess to server-side aspects of software program(s) that are identifiedin request(s) 118B via connection 120B and one or more of connections122A-122N, depending on which of the servers 108A-108N execute theserver-side aspects of the software program(s), and so on.

In accordance with some example embodiments, clients 102A-102M arecapable of accessing Web sites hosted by servers 108A-108N, so thatclients 102A-102M may access information that is available via the Websites. Such information may include documents (e.g., Web pages, images,video files, etc.), output of executables, or any other suitable type ofinformation. The Web pages may be provided as hypertext markup language(HTML) documents and objects (e.g., files) that are linked therein, forexample. In accordance with these embodiments, the information may beprovided by server-side aspects of software programs that are identifiedin requests that are provided by clients 102A-102M.

Clients 102A-102M include respective operating system (OS) modules116A-116M and test agent modules 126A-126M. Each of the OS modules116A-116M executes an operating system on the client that includes thatOS module. Each operating system performs operations which may includebut are not limited to managing computer hardware resources, providingservices for execution of software programs, etc. on the correspondingclient. Examples of an operating system include but are not limited toBerkeley Software Distribution™ (BSD), developed and distributed by theComputer Systems Research Group (CSRG) of the University of California,Berkeley, or descendants thereof; Linux developed and distributed underthe GNU Project; Mac OS® developed and distributed by Apple Inc.,Microsoft Windows® developed and distributed by Microsoft Corporation;and UNIX™ developed and distributed by AT&T.

Each of the test agent modules 126A-126M is configured to execute arespective test agent in accordance with the program testinginstructions 128 received from auto-allocation program tester 106. Eachtest agent performs a respective subset of the operations that imposethe specified workload during the testing of the software program(s).The test agent modules 126A-126M perform client-side aspects of thesoftware program(s).

Auto-allocation program tester 106 is a processing system that testssoftware program(s), which may be deployed on any one or more of clients102A-102M. Auto-allocation program tester 106 provides program testinginstructions 128 regarding testing of software program(s) among clients102A-102M via one or more of connections 120A-120M. For example,auto-allocation program tester 106 may provide the program testinginstructions 128 to only those clients that are to participate intesting of the software program(s). In another example, auto-allocationprogram tester 106 may provide the program testing instructions 128 toall of the clients 102A-102M, regardless whether the clients 102A-102Mare to participate in the testing of the software program(s). Inaccordance with this example, each of the clients 102A-102M may becapable of determining whether it is to participate in the testing ofthe software program(s) based on the instructions 128. Auto-allocationprogram tester 106 receives access to client-side aspects of thesoftware program(s) via connection 126 and one or more of connections128A-28M. Auto-allocation program tester 106 receives access toserver-side aspects of the software program(s) via connection 126 andone or more of connections 122A-122N.

In accordance with example embodiments, auto-allocation program tester106 automatically determines how many of the clients 102A-102M are to beallocated for testing a designated software program based on a specifiedworkload that is to be imposed during the testing. Auto-allocationprogram tester 106 automatically deploys the designated software programon the determined number of the clients 102A-102M and executes the testwith respect to those clients. For instance, auto-allocation programtester 106 may automatically deploy an instance of the designatedsoftware program on each of the determined number of the clients102A-102M. Auto-allocation program tester 106 may select among variousmodes of the test to achieve the designated workload and/or to testvarious capabilities of the software program. For instance, a functionalmode may correspond to a first workload, a performance mode maycorrespond to a second workload, a stress mode may correspond to a thirdworkload, a scalability mode may correspond to a fourth workload, and soon. Some example techniques for automatically allocating clients fortesting a software program are discussed below with reference to FIGS.2-8.

Each of servers 108A-108N is a processing system that is capable ofcommunicating with clients 102A-102M and auto-allocation program tester106. Each of servers 108A-108N is configured to perform operations withregard to the software program to impose a respective proportion of thespecified workload. Although servers 108A-108N are described herein asbeing processing systems, it will be recognized that any one or more ofservers 108A-108N may be implemented as a virtual machine.

Servers 108A-108N include respective operating system (OS) modules112A-112N and web server modules 114A-114N. Each of the OS modules112A-112N executes an operating system on the server that includes thatOS module. Each operating system performs operations which may includebut are not limited to managing computer hardware resources, providingservices for execution of software programs, etc. on the correspondingclient.

Each of the web server modules 114A-114N provides access to softwareprogram(s) that are identified by requests (e.g., requests 118A-118N)received from clients 102A-102M and/or that are identified by programtesting instructions (e.g., program testing instructions 128) receivedfrom auto-allocation program tester 106. Upon receiving a request orinstruction that identifies a software program, web server modules114A-114N may access store 110 via respective connections 122A-122N andconnection 124 to write information thereto and/or to read informationtherefrom in accordance with operations that are defined by the softwareprogram.

In accordance with some example embodiments, servers 108A-108N host websites via which information may be accessed by clients 102A-102M and/orby auto-allocation program tester 106. Such information may includedocuments (e.g., Web pages, images, video files, etc.), output ofexecutables, or any other suitable type of information. In accordancewith these embodiments, the information may be provided by softwareprogram(s) that are identified in requests received from clients102A-102M and/or in program testing instructions 128 received fromauto-allocation program tester 106.

Auto-allocation program tester 106 may be implemented in various ways toautomatically allocate clients for testing a software program, includingbeing implemented in hardware, software, firmware, or any combinationthereof. For example, auto-allocation program tester 106 may beimplemented as computer program code configured to be executed in one ormore processors. In another example, auto-allocation program tester 106may be implemented as hardware logic/electrical circuitry. In anembodiment, auto-allocation program tester 106 may be implemented in asystem-on-chip (SoC). Each SoC may include an integrated circuit chipthat includes one or more of a processor (e.g., a microcontroller,microprocessor, digital signal processor (DSP), etc.), memory, one ormore communication interfaces, and/or further circuits and/or embeddedfirmware to perform its functions.

Store 110 stores information regarding software programs. Suchinformation may include but is not limited to configuration information,metadata, and content regarding the software programs. Configurationinformation is information that specifies how a software program is tobe configured. Metadata may include any suitable information regarding asoftware program, such as information that indicates a network bindingregarding the software program. Content may include documents (e.g., Webpages, images, video files, etc.), output of executables, etc. Store 110may be any suitable type of store. One type of store is a database. Forinstance, store 110 may be a relational database, an entity-relationshipdatabase, an object database, an object relational database, anextensible markup language (XML) database, etc. Store 110 is shown inFIG. 1 to be external to servers 108A-108N for illustrative purses andis not intended to be limiting. It will be recognized that store 110 ora portion thereof may be included in any one or more of servers108A-108N. For instance, store 110 or a portion thereof may bedistributed across two or more of servers 108A-108N.

Testing that is performed by auto-allocation program tester 106 isdescribed herein as being performed with respect to a plurality ofclients 102A-102M for illustrative purposes and is not intended to belimiting. It will be recognized that the testing that is performed byauto-allocation program tester 106 may be performed with respect to asingle client. In fact, auto-allocation program tester 106 may beincorporated into the single client. It will be recognized that theaforementioned testing need not necessarily be performed via a network,such as network 104. Furthermore, computer system 100 need notnecessarily include user systems servers 108A-108N. For instance, theaforementioned testing may be performed entirely in an on-premisesenvironment, which may not include servers 108A-108N.

FIGS. 2-4 depict flowcharts 200, 300, and 400 of example methods forautomatically allocating clients for software program testing inaccordance with embodiments. Flowcharts 200, 300, and 400 may beperformed by auto-allocation program tester 106 of computer system 100shown in FIG. 1, for example. For illustrative purposes, flowcharts 200,300, and 400 are described with respect to an auto-allocation programtester 500 shown in FIG. 5, which is an example of an auto-allocationprogram tester 106, according to an embodiment. As shown in FIG. 5,auto-allocation program tester 500 includes determination logic 502,allocation logic 504, execution logic 506, model logic 508, monitoringlogic 510, collection logic 512, aggregation logic 514, reporting logic516, filtering logic 518, and selection logic 520. Further structuraland operational embodiments will be apparent to persons skilled in therelevant art(s) based on the discussion regarding flowcharts 200, 300,and 400.

As shown in FIG. 2, the method of flowchart 200 begins at step 202. Instep 202, a number of clients to be allocated in a targeted environmentfor participation in a test of a software program is automaticallydetermined based on a specified cumulative workload. In an exampleimplementation, determination logic automatically determines the numberof clients to be allocated in the targeted environment.

At step 204, the number of clients is automatically allocated in thetargeted environment to perform operations that impose the cumulativeworkload against the targeted environment. Each client of the number ofclients imposes a respective proportion of the cumulative load againstthe targeted environment. In an example implementation, allocation logic504 automatically allocates the number of clients in the targetedenvironment.

At step 206, the test of the software program is executed with respectto the number of clients such that each client of the number of clientssimulates use of the software program by at least one user. Forinstance, a test agent may be deployed on each client of the number ofclients to perform a respective subset of the operations, to coordinateperformance of the respective subset of the operations with others ofthe clients, and to collect test results based on performance of therespective subset of the operations. In an example implementation,execution logic 506 executes the test of the software program withrespect to the number of clients. For instance, execution logic 506 maydeploy the test agents on respective test agent modules 116A-116N.

In an example embodiment, step 202 includes automatically determining anumber of clients to be allocated in the targeted environment for eachof a plurality of modes of the test based on the specified cumulativeworkload. In accordance with this embodiment, step 204 includesautomatically allocating the respective number of clients in thetargeted environment for each mode. In one aspect of this embodiment,step 206 includes executing the test in a first mode of the plurality ofmodes with respect to a first number of the clients and executing thetest in a second mode of the plurality of modes with respect to a secondnumber of the clients. The second number is different from the firstnumber.

In another aspect of this embodiment, the plurality of modes includes afunctional mode and at least one of a performance mode, a stress mode,or a scalability mode. The test in the functional mode is configured todetermine whether the software program generates an expected output. Thetest in the performance mode is configured to determine an amount oftime that the software program takes to perform an operation uponreceipt of an instruction to perform the operation. The test in thestress mode is configured to determine a capability of the softwareprogram to remain operational (e.g., continue to provide an expectedoutput) in response to an increasing load over a period of time. Thetest in the scalability mode is configured to incrementally increase aworkload that is associated with the test and to determine, at eachincremental increase, a number of clients that are to be allocated in atargeted environment to perform operations that impose the workloadagainst the targeted environment

In some example embodiments, one or more steps 202, 204, and/or 206 offlowchart 200 may not be performed. Moreover, steps in addition to or inlieu of steps 202, 204, and/or 206 may be performed. For instance, in anexample embodiment, the method of flowchart 200 includes selectingcharacteristics of the test that are to be monitored based on a type ofthe software program. In an example implementation, selection logic 520selects the characteristics of the test that are to be monitored. Inaccordance with this embodiment, the method of flowchart 200 furtherincludes aggregating values of the characteristics based on the type ofthe software program. In an example implementation, aggregation logic514 aggregates the values of the characteristics. In an aspect of thisembodiment, step 204 includes automatically allocating a first subset ofthe number of clients in an on-premises environment that is included inthe targeted environment and automatically allocating a second subset ofthe number of clients in a cloud environment that is included in thetargeted environment. In accordance with this aspect, performance of thesoftware program is monitored with respect to the first subset and thesecond subset simultaneously. In an example implementation, monitoringlogic 510 monitors the performance of the software program with respectto the first subset and the second subset simultaneously.

In another example embodiment, the method of flowchart 200 includescollecting a first subset of test results that are based on performanceof the operations that impose the cumulative workload in response toeach test result in the first subset indicating an error. In accordancewith this embodiment, the method of flowchart 200 further includes notcollecting a second subset of the test results in response to each testresult in the second subset not indicating an error. In an exampleimplementation, collection logic 512 collects the first subset of thetest results and does not collect the second subset of the test results.

In yet another example embodiment, the method of flowchart 200 includesthe steps that are shown in flowchart 300 of FIG. 3. As shown in FIG. 3,the method of flowchart 300 begins at step 302. In step 302, testresults from each client are aggregated into a respective file. In anexample implementation, aggregation logic 514 aggregates the testresults from each client into a respective file.

At step 304, the files are aggregated into a database. In an exampleimplementation, aggregation logic 514 aggregates the files into adatabase that is included in store 110.

At step 306, data that are included in the files are aggregated into amemory to generate a report regarding performance of the softwareprogram during the test. For instance, a dynamic interval may be used toaggregate a predetermined fixed number of the data that are included inthe files into the memory. The dynamic interval indicates a period oftime between temporally adjacent data. In an example implementation,aggregation logic 514 aggregates the data that are included in the filesinto a memory that is included in store 110.

In still another example embodiment, the method of flowchart 200includes the steps that are shown in flowchart 400 of FIG. 4. As shownin FIG. 4, the method of flowchart 400 begins at step 402. In step 402,a report is generated that includes test results describing performanceof the software program during the test. In an example implementation,reporting logic 516 generates the report that includes the test results.

At step 404, information regarding the test results is automaticallyaggregated from the report to provide aggregated information thatidentifies failures. In an example implementation, aggregation logic 514automatically aggregates the information regarding the rest results formthe report to provide the aggregated information that identifies thefailures.

At step 406, the aggregated information is filtered with respect tocommon characteristics among the failures to determine whether adesignated failure that occurs with respect to a specified client isincluded in the failures. In an example implementation, filtering logic518 filters the aggregated information with respect to the commoncharacteristics to determine whether the designated failure that occurswith respect to the specified client is included in the failures.

In an aspect of this embodiment, a determination may be made that asubset of the aggregated information that identifies a specified failurecorresponds to a designated common characteristic, which corresponds tothe designated failure, to an extent that exceeds a threshold. In anexample implementation, determination logic 502 determines that thesubset of the aggregated information that identifies the specifiedfailure corresponds to the designated common characteristic to an extentthat exceeds the threshold. In accordance with this aspect, adetermination is made that the designated failure is included in thefailures in response to determining that the subset of the aggregatedinformation that identifies the specified failure corresponds to thedesignated common characteristic to the extent that exceeds thethreshold. In an example implementation, determination logic 502determines that the designated failure is included in the failures.

It will be recognized that auto-allocation program tester 500 may notinclude one or more of determination logic 502, allocation logic 504,execution logic 506, model logic 508, monitoring logic 510, collectionlogic 512, aggregation logic 514, reporting logic 516, filtering logic518, and/or selection logic 520. Furthermore, auto-allocation programtester 500 may include modules in addition to or in lieu ofdetermination logic 502, allocation logic 504, execution logic 506,model logic 508, monitoring logic 510, collection logic 512, aggregationlogic 514, reporting logic 516, filtering logic 518, and/or selectionlogic 520.

In an example embodiment, a framework may be employed to dynamicallyscale deployment of the test among clients 102A-102M. For instance, theframework may be capable of deploying the test on a single client,hundreds of clients, thousands of clients, etc. to test characteristicsof the software program under a variety of conditions (e.g., peak load,expected load, etc.). The framework may use a cloud platform toprovision and/or de-provision clients 102A-102M on demand. In accordancewith this embodiment, the framework as a service may launch one or moreof clients 102A-102M based on the specified cumulative workload. Forinstance, ten (or any other suitable number) of the clients 102A-102Mmay be may be sufficient for performing the test in the performance modeor the stress mode for one test pass of a single server deployment. Theframework may automatically provision more of the clients 102A-102M ondemand and/or de-provision client(s) when the test pass status is“closed”, meaning that the test is complete. The number of clients thatare provisioned for the test is based on the configuration of each testpass configuration. By using a scalable automatic deployment, theframework may be used to test against any suitable size of the softwareprogram.

In another example embodiment, a framework may be employed to performmultiple levels of aggregation and filtering with respect to testresults that are generated during testing of the software program. Forexample, testing an enterprise application running on a large scaledeployment in a stress mode may involve simulating thousands of clientsin parallel, which may correspond to thousands of transactions persecond depending on the capacity of the enterprise application. Inaccordance with this example, such testing may last for hours or days.Hundreds, thousands, or millions of data entries may be collected pertest pass. A substantial amount of support data (e.g., server state perrole, server side error, event logs, etc.) may be collected in additionto the aforementioned data entries for processing (e.g., generatingreports).

Accordingly, multiple aggregations and filtering operations may beperformed on different levels of the framework. For instance, anassumption may be made that a most detailed report that an end usermight want is a curve chart of specified performance counts in a timetrend (e.g., the CPU usage for each service role, the failure ratethroughout the execution period, etc.). The aggregations may beperformed using a dynamic interval to generate a fixed number ofaggregated data. The fixed number of aggregated data may be used toprovide a relatively high definition chart for investigation using aleast amount of data as possible. For example, assume for purposes ofillustration that 400 points of aggregated data are desired. If eachtest case execution duration is 20 minutes, the aggregation interval maybe 20*60/400=3 seconds. The test results that correspond to each 3second interval may be aggregated into a single respective data entry(i.e., one data entry for each interval). Testing an enterprise-levelapplication in a performance mode or a stress mode may provide thousandsof test results in a 3 second interval. Having one data entry per fixedinterval may assist in limiting a quantity of the test results to amanageable level.

Aggregation may be performed on a variety of test data types. Examplesof test data types include but are not limited to test results for eachtest case and step iteration, test failures for each test case and stepiteration, performance factors monitored from a client-side or aserver-side, server error logs monitored from the server side, eventlogs monitored from the client side and the server side, etc. The testresults may be filtered based on any suitable criteria. For example,test results may be collected only with regard to failures of thesoftware program for non-performance sensitive test results. Inaccordance with this example, test results regarding passed validationsteps may not be included in the collected test results. In anotherexample, event logs may be filtered by query (e.g., Even XPath query).In yet another example, server error logs may be filtered by errorlevel.

Aggregation may be performed on a variety of levels. For instance, anindividual client may aggregate the test results, client-sideperformance counters, and event logs locally in memory. The aggregationmay be based primarily on the timeline, for example. Server-sideaggregation may be executed when the test case execution is completedand all of the clients' aggregated test results are uploaded. Theaggregation may be based on the timeline and/or roles (e.g., client,frontend, mid-layer, and database).

In yet another example embodiment, monitoring of a software program isscalable among various tiers of the software program. For instance, thesoftware program may have hundreds of individual server instances,servicing in different service roles and/or in different tiers. Anarchitecture for the software program may include a frontend(presentation layer), a mid-layer (business layer), a database (datalayer), and a client if the software program is a rich client solution.A scalable monitoring solution may be employed to collect and aggregatedifferent types of state data from different roles and/or clients. Themonitoring solution may be used to apply a different monitoring settingon each server role depending on the nature of that server role.Monitoring may be performed with respect to a variety of types ofmonitoring targets. Examples of a monitoring target include but are notlimited to a performance factor (e.g., performance counters of anoperating system (OS), custom performance counters logged by test code,etc.), an event (e.g., event of an OS), and a log. Monitoring may beperformed with respect to multiple layers of a monitoring target. Forinstance, the monitoring configuration may be grouped by role to supportan N-layer software application, where N is a positive integer. Multipleclients may be supported in each role (e.g., 20 front end serverinstances). Each test scenario may have its own monitoring scenario forthe various test roles.

In still another example embodiment, heterogeneous platform monitoringis used during testing of the software program. For instance, thesoftware program may be deployed on multiple platforms (e.g., aMicrosoft Windows® platform and a non-Microsoft Windows platform). Thesoftware program may be deployed in multiple locations (e.g.,on-premises and in the cloud). If the software program includes a richclient, the client may be run on a different platform from the test.Different test environments may be deployed on different platforms. Theframework is capable of supporting the software program being deployedon multiple types of predetermined platforms. The framework can beextended to support platforms in addition to the predetermined platformsby adding plug-ins that implement a suitable interface (e.g., anIMonitor interface).

In an example embodiment, a framework may be employed that is capable ofscaling with respect to development of the test for testing the softwareprogram. In a conventional framework, development of the performance,stress, and scalability test cases is isolated from development of thefunctional test case. Accordingly, a substantial amount of effort isemployed to obtain code coverage in the performance, stress, andscalability test cases that is similar to the code coverage obtained inthe functional test case. In accordance with the aforementionedembodiment, a functional test case may be converted into a performance,stress, and/or scalability test case at a relatively low cost. With astandard test case development tool, for example, the functional testcase may be converted to a different mode (e.g., performance, stress, orscalability), automatically with relatively little (e.g., no) additionaldevelopment cost.

A script-like execution shell may help to fully reuse existingfunctional test codes. An example of such an execution shell isrepresented using XML code below for illustrative purposes.

<StepGroups>  <Setup FailureAction=“FailAndStop”>   <VariableType=“@(Common).EDMMeta” Name=“Sserv1”>    <ArgType=“string”>WoltersKluwer/SalesTaxRates</Arg>   </Variable>   <StepMethod=“@(Common).EDMMeta.get_QueryBuilderCollection” Object=“Sserv1”Result=“Sugc1”/>  </Setup>  <Regular>   <StepGroupFailureAction=“FailAndStop”>    <StepMethod=“@(Common).QueryBuilderCollection.    GetRandomUrl”Object=“Sugc1” Result=“Surl1”/>    <Variable Name=“$rq1”Type=“@(Common).Request”>     <Arg Type=“string”>Surl1</Arg>   </Variable>    <Step Method=“@(Common).Request.GetReponse”Result=“Srp1” Object=“$rq1” Title=“Query WoltersKluwer SalesTaxRates”PerfStep=“true” >     <Arg Type=“bool”>false</Arg>    </Step>    <StepMethod=“@(Common).Response.Read” Object=“Srp1” />    <StepMethod=“@(Perf).Utility.ValidateResponse” Expect=“true”>     <ArgType=“@(Common).Request”>$rq1</Arg>     <ArgType=“@(Common).Response”>Srp1</Arg>     <ArgType=“string”>$E2EHARNESS.TestType</Arg>    </Step>   </StepGroup> </Regular> </StepGroups>

For instance, linear code (e.g., such as linear .Net code) may betranslated into the above-recited XML test case script. The functionaltest code may be separated among setup steps, regular steps, and cleanupsteps. The setup steps are performed before the regular steps. Theregular steps are performed before the cleanup steps. The codecorresponding to the regular steps may be separated among multiplescenarios. The multiple scenarios may be performed with respect to eachthread of the test. An execution weight may be assigned to each type ofsteps (i.e., setup, regular, cleanup, and scenario). For each of thescenario steps, test results are collected only if the step is sensitiveto performance. The scenario steps for which results are not collectednevertheless may affect the test results. Thus, a report that isgenerated in response to performance of the scenario steps reflects theactual performance of the software program. The prerequisite andvalidation test code does not affect the test results. This also mayreduce an amount of the test results that are aggregated and uploaded tostore 110.

In another example embodiment, runtime resources are automaticallyoptimized. For instance, the stress and scalability modes may generate asubstantial amount of data including test results, logs, and monitoringresults from clients. In accordance with this embodiment, the frameworkis capable of automatically determining when and/or how to compute,aggregate, upload, and dispose of the test results to avoid overloadingresources, such as the CPU, memory, and network and/or to avoidaffecting the performance results of the test code. For instance, theframework uploading results may substantially affect the response timeof the performance test if the test is input/output (I/O) sensitive.Because the framework has client-side monitoring, the framework hasknowledge of the CPU and I/O status of each client. Accordingly, theframework may determine when and/or how to perform operations.Thresholds may be established for the CPU, memory, and networkresources. For instance, a CPU utilization threshold may be set at 85%,a memory threshold may be set at 75%, and a network threshold may be setat 50%. These thresholds are provided for illustrative purposes and arenot intended to be limiting. The thresholds may be set at any suitablerespective values. When the memory threshold is reached, the testresults may be removed from the memory. Data may be uploaded from a diskonly in response to test execution being completed or terminated. Inaccordance with this embodiment, the test results may beself-descriptive and may be uploaded order-less (i.e., without regard toorder).

In yet another example embodiment, the test of the software program maybe implemented to be operation controllable. For instance, the softwareprogram may have a capacity on a specified deployment setting. Thecapacity may be determined by an estimated capacity plan (e.g., we planto support X concurrent users sending Y operations per second) or aresource limitation (e.g., we can support X concurrent users sending Yoperations per second on a specified deployment to provide a positiveuser experience). A performance test or a stress test that generates aworkload that exceeds the capacity may cause the test results to becomemeaningless. Operation controls may be applied at the framework level tocause the test to be executed at a desired rate. The test may beexecuted in a different operational level to identify the capacity orbottleneck of a specified server deployment. The framework may havebuilt-in operation control to enable testing to achieve the desiredoperational level.

In still another example embodiment, a framework is employed thatsupports multiple test modes for a common test. In accordance with thisembodiment, different reports may be generated for the various modes ofthe test with little or no extra cost. Following is example XML codethat illustrates how the various modes of the test may be defined:

  <!--Goal 200 OPS, 8+1 test machine--> <TestConfigGroup Name=“TestSVC”DelayInterval =“60” >  <TestConfig TestType=“Perf” StepDelay=“10”>  <OPSConfig OPS1 =“25”/>   <ThreadPool ThreadCount=“300”PoolCount=“1”/>  </TestConfig>  <TestConfig TestType=“Stress”StepDelay=“120”>   <OPSConfig OPS1 =“35”/>   <ThreadPoolThreadCount=“300” PoolCount=“1”/>  </TestConfig>  <TestConfigTestType=“Scale” StepDelay=“10”>   <OPSConfig OPS1 =“1” OPS2=“35”/>  <ThreadPool ThreadCount=“300” PoolCount=“1”/>  </TestConfig> <TestConfig TestType=“STPerf” StepDelay=“10”>   <OPSConfig OPS1 =“10”/>  <ThreadPool ThreadCount=“300” PoolCount=“1”/>  </TestConfig> <TestConfig TestType=“Custom” StepDelay=“1”>   <OPSConfig OPS1 =“1”/>  <ThreadPool ThreadCount=“1” PoolCount=“1”/>  </TestConfig></TestConfigGroup>

For a single thread performance mode, a single thread mode runs throughthe regular steps for a limited iteration to identify performance of aparticular code path. For a load performance mode, multiple thread modesrun through the regular steps at a specified operations-per-second toidentify performance under a particular load. For a stress performancemode, a stress mode runs through the regular steps at a specifiedoperations-per-second over a relatively long duration to identify thesystem availability under an extreme load. A phase may be added at theend of execution to detect whether the software program goes back to itsnormal state after the stress run. For scalability performance mode,multiple threads run through the regular steps, and the operationcontrol gradually increases from a relatively low operations-per-secondto a relatively high operations-per-second to identify the systemcapacity of a particular service deployment. For a custom performancemode, any suitable combination of operations per second control,runtime, and thread control configuration may be utilized.

In an example embodiment, multiple test scenarios are supported. Basedon the software program specification, different test scenarios may havedifferent test goals with regard to performance, stress, andscalability. For instance, activation of a software program may takesubstantially less load than servicing the software program. Inaccordance with this embodiment, the framework is capable of applyingdifferent test configurations on different user scenarios and/ordifferent performance goals on the reporting. For example, each testscenario may have its own TestConfigGroup, though the test configurationshown in the example XML code above is applied only to test scenarioTestSVC for purposes of illustration. Each test scenario may have itsown monitoring configuration. Based on the test scenario, theauto-allocation program tester 106 may decide which kinds of theperformance counter and information is likely to be useful duringinvestigation (and therefore is to be collected). Each test scenario mayhave its own performance goal during the reporting.

In another example embodiment, a scalable test result database isemployed. Although multiple levels of aggregation may be performed onthe client side, a relatively substantial amount of data may be uploadedto the database. The database is configured to be scalable as a testreport storage. For instance, the database may be capable of storingresults and/or reports from hundreds of test passes for furtherinvestigation (e.g., regression and performance trend investigation). Inaddition to solutions such as database table partitioning and databaseclustering, other optimization techniques may be performed on a singledatabase level based on the usage of the test data.

FIG. 6 is a block diagram of an example schema 600 that may be used todefine a structure of a database that is included in a store 110 shownin FIG. 1 in accordance with an embodiment. Schema 600 defines aplurality of types that are represented using a plurality of respectivedata tables 602A-602M. Data table 602A represents a TestResults type;data table 602B represents a CounterState type; data table 602Crepresents an ErrorLog type; data table 602D represents a TestCase type;data table 602E represents a TestPass type; data table 602F represents aTestMachine type; data table 602G represents a LogicalMachine type; datatable 602H represents a PhysicalMachine type; data table 602I representsa Counter type; data table 602J represents a ReportCounter type; datatable 602K represents a Report type; data table 602L represents aReportSimple type; and data table 602M represents an ExecutionResulttype. Each of the data tables 602A-602M includes respective parametershaving values that describe the respective type that is represented bythat data table. For instance, data table 602H includes aPhysicalMachineGuid parameter and a HardwareInfo parameter, the valuesof which describe the PhysicalMachine type.

Data tables 602A-602C are associated with a runtime group 604; datatables 602D-602I are associated with a definition group 606; and datatables 602J-602M are associated with a reporting group 608. Data tables602A-602C in the runtime group 604 store runtime raw test results thatare obtained with regard to testing a software program. For instance,the raw test results may include information that is received from oneor more of the test agent modules 116A-116N and/or information that isreceived from auto-allocation program tester 106 (or one or morecomponents thereof). Data tables 602D-602I in the definition group 606store information that is utilized by data tables 602A-602C of theruntime group 604 and data tables 602J-602M of the reporting group 608.Data tables 602J-602M of the reporting group 608 stores intermediateaggregated report data that may be utilized by a reporting website, forexample. For instance, the intermediate aggregated report data may beaggregated at multiple levels (e.g., aggregated into files, thenaggregated into a database, and then aggregated into a memory togenerate a report regarding performance of the software program).

References 610-618 from data tables 602A-602C in the runtime group 604point to data tables 602D-602I of the definition group 606. Morespecifically, references 610, 612, and 615 point from data table 602A torespective data tables 602D, 602E, and 602F. References 614, 616, and618 point from data table 602B to respective data tables 602E, 602G, and602I. References 611, 613, and 617 point from data table 602C torespective data tables 602D, 602E, and 602G.

In the definition group 606, references 619 and 620 point from datatable 602F to respective data tables 602E and 602G. Reference 621 pointsfrom data table 602G to data table 602H.

References 622-627 from data tables 602K and 602M in the reporting group608 point to data tables 602D, 602E, 602G, and 602I of the definitiongroup 606. More specifically, references 622, 624, 626, and 627 pointfrom data table 602K to respective data tables 602D, 602E, 602G, and602I. References 623 and 625 point from data table 602M to respectivedata tables 602D and 602E.

In the reporting group 608, reference 628 points from data table 602J todata table 602K. Reference 629 points from data table 602L to data table602K. None of the references 610-629 point from the definition group606. None of the references 610-629 point into the runtime group 604.None of the references 610-629 point into the reporting group 608.

In an example embodiment, reporting of the test results is scalable. Forinstance, auto-allocation program tester 106 of FIG. 1 orauto-allocation program tester 500 of FIG. 5 may be capable ofgenerating a relatively simple report showing summary results and alsomay be capable of generating a relatively detailed report showingrelevance of a designated performance factor (e.g., response time) of adesignated test case on a designated test client and another performancefactor (e.g., SQL Server® process CPU usage) on a specified servermachine instance of some server role. The report may be returned to theend user in a timely manner. Accordingly, generation of the report mayweigh aggregation of the data (which improves the reporting speed) andpreservation of the raw data (which increases diversity of the report).

As described above, multiple levels of aggregation may be applied toeach phase of the execution and reporting. The final level ofaggregation may be a report generator based on the aggregatedinformation uploaded by clients 102A-102M of FIG. 1 and monitoring logic510 of FIG. 5 and aggregated by role, machine instance, and timeline. Inan example, two general types of reports are optimized during thisprocess. The first type of report includes a summary result of theexecution (e.g., a table that shows the summary result of the testpass). The second type of report includes a timeline based chart. Anyselected set of performance factors may be overlapped in the chart toshow relevance. A set of object model APIs may be provided to access thedatabase to generate the reports. A most detailed raw test log may becompressed and uploaded to storage of a cloud platform, for example, byeach of the clients 102A-102M so that the logs may be used for furtherinvestigation.

In another example embodiment, auto-allocation program tester 106 ofFIG. 1 is capable of performing auto-triage with respect to the testresults. For instance, an auto-triage operation may be performed inresponse to performance of the test of the software program in theperformance mode or stress mode. Executing the test for an extendedperiod (e.g., multiple days) may result in generation of thousands oferror test results and server-side error logs. The auto-triage operationmay be performed to summarize the test results and logs, to collaboratewith failure information identified in previous run(s), and/or topattern match in order to automatically determine which failures areknown and which failures were previously unknown. The auto-triageoperation may lower the investigation cost and/or identify thepreviously unknown failures for reference in subsequently generatedreports.

Each error may be identified as being either a known error or a newerror. The known errors that are active and that are associated with atest case may be identified. Information regarding the active knownerrors may be retrieved (e.g., from store 110). Based on the aggregatedresults per test case, a similarity between known errors and failuresmay be determined. Based on the similarity, the new errors may beautomatically opened and/or the previously known errors may be signedoff. The auto-triage operation may be performed using client-side errorlog(s) and/or server-side error log(s), examples of which are discussedbelow with reference to respective FIGS. 7 and 8.

FIG. 7 illustrates an example client-side error log 700 in accordancewith an embodiment. Client-side error log 700 includes an error messagecolumn 702 and a report count column 704. The error message column 702lists some example types of error message that may be provided by anyone or more of clients 102A-102M based on testing the software program.The report count column 704 lists a number of instances of each errormessage that occurs with respect to a test pass that is performed on thesoftware program. For example, 365 instances of the BadGateway errormessage are provided with regard to the test pass that corresponds toclient-side error log 700. In another example, 9506 instances of theBadRequest error message are provided with regard to the test pass.

FIG. 8 illustrates an example server-side error log 800 in accordancewith an embodiment. Server-side error log 800 includes a message column802 and a report count column 804. The message column 802 lists someexample messages that may be provided by auto-allocation program tester106 or 500 based on testing the software program. The report countcolumn 804 lists a number of instances of each message that is providedbased on the test pass. For example, a message of “Object reference notset to an instance of an object” is provided once. In another example, amessage of “The operation has timed out” is provided 6,208 times.

Auto-allocation program tester 106, each of OS modules 112A-112N, eachof web server modules 114A-114N, each of test agent modules 116A-116N,determination logic 502, allocation logic 504, execution logic 506,model logic 508, monitoring logic 510, collection logic 512, aggregationlogic 514, reporting logic 516, filtering logic 518, selection logic520, and flowcharts 200, 300, and 400 may be implemented in hardware,software, firmware, or any combination thereof.

For example, auto-allocation program tester 106, each of OS modules112A-112N, each of web server modules 114A-114N, each of test agentmodules 116A-116N, determination logic 502, allocation logic 504,execution logic 506, model logic 508, monitoring logic 510, collectionlogic 512, aggregation logic 514, reporting logic 516, filtering logic518, selection logic 520, flowchart 200, flowchart 300, and/or flowchart400 may be implemented as computer program code configured to beexecuted in one or more processors.

In another example, auto-allocation program tester 106, each of OSmodules 112A-112N, each of web server modules 114A-114N, each of testagent modules 116A-116N, determination logic 502, allocation logic 504,execution logic 506, model logic 508, monitoring logic 510, collectionlogic 512, aggregation logic 514, reporting logic 516, filtering logic518, selection logic 520, flowchart 200, flowchart 300, and/or flowchart400 may be implemented as hardware logic/electrical circuitry. Forinstance, in an embodiment, one or more of auto-allocation programtester 106, OS modules 112A-112N, web server modules 114A-114N, testagent modules 116A-116N, determination logic 502, allocation logic 504,execution logic 506, model logic 508, monitoring logic 510, collectionlogic 512, aggregation logic 514, reporting logic 516, filtering logic518, selection logic 520, flowchart 200, flowchart 300, and/or flowchart400 may be implemented in a system-on-chip (SoC). The SoC may include anintegrated circuit chip that includes one or more of a processor (e.g.,a microcontroller, microprocessor, digital signal processor (DSP),etc.), memory, one or more communication interfaces, and/or furthercircuits and/or embedded firmware to perform its functions.

FIG. 9 depicts an example computer 900 in which embodiments may beimplemented. Any one or more of the clients 102A-102M, any one or moreof servers 108A-108N, or auto-allocation program tester 106 shown inFIG. 1 (or any one or more subcomponents thereof shown in FIG. 5) may beimplemented using computer 900, including one or more features ofcomputer 900 and/or alternative features. Computer 900 may be ageneral-purpose computing device in the form of a conventional personalcomputer, a mobile computer, or a workstation, for example, or computer900 may be a special purpose computing device. The description ofcomputer 900 provided herein is provided for purposes of illustration,and is not intended to be limiting. Embodiments may be implemented infurther types of computer systems, as would be known to persons skilledin the relevant art(s).

As shown in FIG. 9, computer 900 includes a processing unit 902, asystem memory 904, and a bus 906 that couples various system componentsincluding system memory 904 to processing unit 902. Bus 906 representsone or more of any of several types of bus structures, including amemory bus or memory controller, a peripheral bus, an acceleratedgraphics port, and a processor or local bus using any of a variety ofbus architectures. System memory 904 includes read only memory (ROM) 908and random access memory (RAM) 910. A basic input/output system 912(BIOS) is stored in ROM 908.

Computer 900 also has one or more of the following drives: a hard diskdrive 914 for reading from and writing to a hard disk, a magnetic diskdrive 916 for reading from or writing to a removable magnetic disk 918,and an optical disk drive 920 for reading from or writing to a removableoptical disk 922 such as a CD ROM, DVD ROM, or other optical media. Harddisk drive 914, magnetic disk drive 916, and optical disk drive 920 areconnected to bus 906 by a hard disk drive interface 924, a magnetic diskdrive interface 926, and an optical drive interface 928, respectively.The drives and their associated computer-readable storage media providenonvolatile storage of computer-readable instructions, data structures,program modules and other data for the computer. Although a hard disk, aremovable magnetic disk and a removable optical disk are described,other types of computer-readable storage media can be used to storedata, such as flash memory cards, digital video disks, random accessmemories (RAMs), read only memories (ROM), and the like.

A number of program modules may be stored on the hard disk, magneticdisk, optical disk, ROM, or RAM. These programs include an operatingsystem 930, one or more application programs 932, other program modules934, and program data 936. Application programs 932 or program modules934 may include, for example, computer program logic for implementingauto-allocation program tester 106, each of OS modules 112A-112N, eachof web server modules 114A-114N, each of test agent modules 116A-116N,determination logic 502, allocation logic 504, execution logic 506,model logic 508, monitoring logic 510, collection logic 512, aggregationlogic 514, reporting logic 516, filtering logic 518, selection logic520, flowchart 200 (including any step of flowchart 200), flowchart 300(including any step of flowchart 300), and/or flowchart 400 (includingany step of flowchart 400), as described herein.

A user may enter commands and information into the computer 900 throughinput devices such as keyboard 938 and pointing device 940. Other inputdevices (not shown) may include a microphone, joystick, game pad,satellite dish, scanner, or the like. These and other input devices areoften connected to the processing unit 902 through a serial portinterface 942 that is coupled to bus 906, but may be connected by otherinterfaces, such as a parallel port, game port, or a universal serialbus (USB).

A display device 944 (e.g., a monitor) is also connected to bus 906 viaan interface, such as a video adapter 946. In addition to display device944, computer 900 may include other peripheral output devices (notshown) such as speakers and printers.

Computer 900 is connected to a network 948 (e.g., the Internet) througha network interface or adapter 950, a modem 952, or other means forestablishing communications over the network. Modem 952, which may beinternal or external, is connected to bus 906 via serial port interface942.

As used herein, the terms “computer program medium” and“computer-readable medium” are used to generally refer to media such asthe hard disk associated with hard disk drive 914, removable magneticdisk 918, removable optical disk 922, as well as other media such asflash memory cards, digital video disks, random access memories (RAMs),read only memories (ROM), and the like. Such computer-readable storagemedia are distinguished from and non-overlapping with communicationmedia. Communication media typically embodies computer-readableinstructions, data structures, program modules or other data in amodulated data signal such as a carrier wave. The term “modulated datasignal” means a signal that has one or more of its characteristics setor changed in such a manner as to encode information in the signal. Byway of example, and not limitation, communication media includeswireless media such as acoustic, RF, infrared and other wireless media.Example embodiments are also directed to such communication media.

As noted above, computer programs and modules (including applicationprograms 932 and other program modules 934) may be stored on the harddisk, magnetic disk, optical disk, ROM, or RAM. Such computer programsmay also be received via network interface 950 or serial port interface942. Such computer programs, when executed or loaded by an application,enable computer 900 to implement features of embodiments discussedherein. Accordingly, such computer programs represent controllers of thecomputer 900.

Example embodiments are also directed to computer program productscomprising software (e.g., computer-readable instructions) stored on anycomputer useable medium. Such software, when executed in one or moredata processing devices, causes a data processing device(s) to operateas described herein. Embodiments may employ any computer-useable orcomputer-readable medium, known now or in the future. Examples ofcomputer-readable mediums include, but are not limited to storagedevices such as RAM, hard drives, floppy disks, CD ROMs, DVD ROMs, zipdisks, tapes, magnetic storage devices, optical storage devices,MEMS-based storage devices, nanotechnology-based storage devices, andthe like.

III. Conclusion

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. It will be apparent to persons skilled in the relevantart(s) that various changes in form and details can be made thereinwithout departing from the spirit and scope of the invention. Thus, thebreadth and scope of the present invention should not be limited by anyof the above-described example embodiments, but should be defined onlyin accordance with the following claims and their equivalents.

What is claimed is:
 1. A method comprising: automatically determining anumber of clients to be allocated in a targeted environment forparticipation in a test of a software program based on a specifiedcumulative workload; automatically allocating the number of clients inthe targeted environment to perform operations that impose thecumulative workload against the targeted environment, each client of thenumber of clients imposing a respective proportion of the cumulativeload against the targeted environment, the automatically allocatingincluding: deploying the software program on the number of clients; andexecuting the test of the software program with respect to the number ofclients such that each client of the number of clients simulates use ofthe software program by at least one user.
 2. The method of claim 1,wherein automatically determining the number of clients to be allocatedcomprises: automatically determining a number of clients to be allocatedin the targeted environment for each of a plurality of modes of the testbased on the specified cumulative workload; wherein automaticallyallocating the number of clients comprises: automatically allocating therespective number of clients in the targeted environment for each mode;and wherein executing the test of the software program comprises:executing the test in a first mode of the plurality of modes withrespect to a first number of the clients; and executing the test in asecond mode of the plurality of modes with respect to a second number ofthe clients, the second number being different from the first number. 3.The method of claim 1, wherein automatically determining the number ofclients to be allocated comprises: automatically determining a number ofclients to be allocated in the targeted environment for each of aplurality of modes of the test based on the specified cumulativeworkload; wherein automatically allocating the number of clientscomprises: automatically allocating the respective number of clients inthe targeted environment for each mode; wherein the plurality of modesincludes at least two of a functional mode, a performance mode, a stressmode, or a scalability mode; wherein the test in the functional mode isconfigured to determine whether the software program generates anexpected output; wherein the test in the performance mode is configuredto determine an amount of time that the software program takes toperform an operation upon receipt of an instruction to perform theoperation; wherein the test in the stress mode is configured todetermine a capability of the software program to remain operational inresponse to an increasing load over a period of time; and wherein thetest in the scalability mode is configured to incrementally increase thecumulative workload and to determine, at each incremental increase, thenumber of the clients that are to be allocated in the targetedenvironment to perform the operations that impose the cumulativeworkload against the targeted environment.
 4. The method of claim 1,wherein executing the test of the software program comprises: deployinga test agent on each client of the number of clients to perform arespective subset of the operations, to coordinate performance of therespective subset of the operations with others of the clients, and tocollect test results based on performance of the respective subset ofthe operations.
 5. The method of claim 1, further comprising:aggregating test results from each client into a respective file of aplurality of files; aggregating the plurality of files into a database;and aggregating data that are included in the plurality of files into amemory to generate a report regarding performance of the softwareprogram during the test.
 6. The method of claim 5, wherein aggregatingthe data comprises: using a dynamic interval to aggregate apredetermined fixed number of the data that are included in theplurality of files into the memory, the dynamic interval indicating aperiod of time between temporally adjacent data.
 7. The method of claim1, further comprising: generating a report that includes test resultsdescribing performance of the software program during the test;automatically aggregating information regarding the test results fromthe report to provide aggregated information that identifies a pluralityof failures; and filtering the aggregated information with respect to aplurality of common characteristics among the plurality of failures todetermine whether a designated failure that occurs with respect to aspecified client is included in the plurality of failures.
 8. The methodof claim 7, further comprising: determining that a subset of theaggregated information that identifies a specified failure of theplurality of failures corresponds to a designated common characteristicof the plurality of common characteristics, which corresponds to thedesignated failure, to an extent that exceeds a threshold; anddetermining that the designated failure is included in the plurality offailures in response to determining that the subset of the aggregatedinformation that identifies the specified failure corresponds to thedesignated common characteristic to the extent that exceeds thethreshold.
 9. The method of claim 1, further comprising: selectingcharacteristics of the test that are to be monitored based on a type ofthe software program; and aggregating values of the characteristicsbased on the type of the software program.
 10. The method of claim 9,wherein automatically allocating the number of clients comprises:automatically allocating a first subset of the number of clients in anon-premises environment that is included in the targeted environment;and automatically allocating a second subset of the number of clients ina cloud environment that is included in the targeted environment; andwherein the method further comprises: monitoring performance of thesoftware program with respect to the first subset and the second subsetsimultaneously.
 11. The method of claim 1, further comprising:collecting a first subset of test results that are based on performanceof the operations that impose the cumulative workload in response toeach test result in the first subset indicating an error; and notcollecting a second subset of the test results in response to each testresult in the second subset not indicating an error.
 12. A systemcomprising: determination logic configured to automatically determine anumber of clients to be allocated in a targeted environment forparticipation in a test of a software program based on a specifiedcumulative workload; allocation logic configured to automaticallyallocate the number of clients in the targeted environment, by deployingthe software program on the number of clients, to perform operationsthat impose the cumulative workload against the targeted environment,each client of the number of clients imposing a respective proportion ofthe cumulative load against the targeted environment; and executionlogic configured to execute the test of the software program withrespect to the number of clients such that each client of the number ofclients simulates use of the software program by at least one user. 13.The system of claim 12, wherein the determination logic is configured toautomatically determining a number clients to be allocated in thetargeted environment for each of a plurality of modes of the test basedon the specified cumulative workload; wherein the allocation logic isconfigured to automatically allocate the respective number of clients inthe targeted environment for each mode; and wherein the execution logicis configured to execute the test in a first mode of the plurality ofmodes with respect to a first number of the clients and to execute thetest in a second mode of the plurality of modes with respect to a secondnumber of the clients, the second number being different from the firstnumber.
 14. The system of claim 12, wherein the determination logic isconfigured to automatically determine a number of clients to beallocated in the targeted environment for each of a plurality of modesof the test based on the specified cumulative workload; wherein theallocation logic is configured to automatically allocate the respectivenumber of clients in the targeted environment for each mode; wherein theplurality of modes includes at least two of a functional mode, aperformance mode, a stress mode, or a scalability mode; wherein the testin the functional mode is configured to determine whether the softwareprogram generates an expected output; wherein the test in theperformance mode is configured to determine an amount of time that thesoftware program takes to perform an operation upon receipt of aninstruction to perform the operation; wherein the test in the stressmode is configured to determine a capability of the software program toremain operational in response to an increasing load over a period oftime; and wherein the test in the scalability mode is configured toincrementally increase the cumulative workload and to determine, at eachincremental increase, the number of the clients that are to be allocatedin the targeted environment to perform the operations that impose thecumulative workload against the targeted environment.
 15. The system ofclaim 12, further comprising: aggregation logic configured to aggregatetest results from each client into a respective file of a plurality offiles, the aggregation logic further configured to aggregate theplurality of files into a database, the aggregation logic furtherconfigured to aggregate data that are included in the plurality of filesinto a memory to generate a report regarding performance of the softwareprogram during the test.
 16. The system of claim 12, further comprising:aggregation logic configured to automatically aggregate informationregarding test results describing performance of the software programduring the test from a report to provide aggregated information thatidentifies a plurality of failures; and filtering logic configured tofilter the aggregated information with respect to a plurality of commoncharacteristics among the plurality of failures to determine whether adesignated failure that occurs with respect to a specified client isincluded in the plurality of failures.
 17. The system of claim 16,wherein the determination logic is further configured to determinewhether a subset of the aggregated information that identifies aspecified failure of the plurality of failures corresponds to adesignated common characteristic of the plurality of commoncharacteristics, which corresponds to the designated failure, to anextent that exceeds a threshold; and wherein the determination logic isfurther configured to determine that the designated failure is includedin the plurality of failures in response to a determination that thesubset of the aggregated information that identifies the specifiedfailure corresponds to the designated common characteristic to theextent that exceeds the threshold.
 18. The system of claim 12, whereinthe allocation logic is configured to automatically allocate a firstsubset of the number of clients in an on-premises environment that isincluded in the targeted environment; wherein the allocation logic isconfigured to automatically allocate a second subset of the number ofclients in a cloud environment that is included in the targetedenvironment; and wherein the system further comprises: monitoring logicconfigured to monitor performance of the software program with respectto the first subset and the second subset simultaneously.
 19. A computerprogram product comprising a computer-readable medium having computerprogram logic recorded thereon for enabling a processor-based system toautomatically allocate clients for testing a software program, thecomputer program logic comprising: a first program logic module forenabling the processor-based system to automatically determine a numberof clients to be allocated in a targeted environment for participationin a test of a software program based on a specified cumulativeworkload; a second program logic module for enabling the processor-basedsystem to automatically allocate the number of clients in the targetedenvironment, by deploying the software program on the number of clients,to perform operations that impose the cumulative workload against thetargeted environment, each client of the number of clients imposing arespective proportion of the cumulative load against the targetedenvironment; and a third program logic module for enabling theprocessor-based system to execute the test of the software program withrespect to the number of clients such that each client of the number ofclients simulates use of the software program by at least one user. 20.The computer program product of claim 19, wherein the first programlogic module includes logic for enabling the processor-based system toautomatically determine a number of clients to be allocated in thetargeted environment for each of a plurality of modes of the test basedon the specified cumulative workload; wherein the second program logicmodule includes logic for enabling the processor-based system toautomatically allocate the respective number of clients in the targetedenvironment for each mode; wherein the plurality of modes includes atleast two of a functional mode, a performance mode, a stress mode, or ascalability mode; wherein the test in the functional mode is configuredto determine whether the software program generates an expected output;wherein the test in the performance mode is configured to determine anamount of time that the software program takes to perform an operationupon receipt of an instruction to perform the operation; wherein thetest in the stress mode is configured to determine a capability of thesoftware program to remain operational in response to an increasing loadover a period of time; and wherein the test in the scalability mode isconfigured to incrementally increase the cumulative workload and todetermine, at each incremental increase, the number of the clients thatare to be allocated in the targeted environment to perform theoperations that impose the cumulative workload against the targetedenvironment.